Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-1560

Description

• Here are some details about my PR, (No UI Changes):

The PR addresses airflow integration with AWS Dynamodb.

Currently there is no hook to interact with DynamoDb for reading or writing items (single or batch insertions). To get started, we want to push data in DynamoDB using airflow jobs (scheduled daily). Idea is to read aggregates from Hive and push in DynamoDB (write data job will run everyday to make this happen). First we want to create DynamoDB hooks (this PR addressed the same) and then create operator to move data from Hive to DynamoDB.
http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#service-resource

Something to think about - Performance
I think airflow's generic approach for reading all data from source and then transferring to the destination by either storing source data in memory or disk is not the best optimized approach (in cases of big data with potential performance issues). When I started my initial design - I categorized it into 2 use cases a) small data b) big data.. The operators written in airflow code right now mostly handle small data (so does this PR). My approach would be to start with this operator (small data) and then handle big data use case if we see performance issues. I have some thoughts on how we can handle big data operations (using batching mechanism). For instance:

1. Fetch all distinct id's (primary key) from source
2. Create N buckets and assign bucket number to all the id's
3. For each bucket : fetch data from source and then insert into destination.

Plan to handle this in subsequent PR.

Something to think about - Other ways of transferring data from hive to dynamodb

There can be multiple ways for this data transfer to occur and here are some of the thoughts on current design decisions (KISS - keep it simple and sweet):

1. Lambda function : You can write a lambda function to transfer data and trigger the function using AwsLambdaHook ([AIRFLOW-1571] Add AWS Lambda Hook #2570) but lambda lifetime is 5 mins and if the data transfer takes longer lambda will die.

2. Amazon Ecosystem (Amazon EMR, Big Data Pipeline, AWS Glue): Not to forget adding these systems create additional complexities (aws dependencies, aws limits, system complexities, IAM issues, airflow integrations with additonal dependencies and ofcourse $$)

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html

3. Not to forget insertions in dynamodb are tricky because of WCU's and boto handles the insertions gracefully by automatically throttling insertions (buffering etc)

Tests

• My PR adds the following unit tests :

1. test_aws_hook.py
   adds mock_dynamodb2 from moto (https://github.com/spulec/moto). Wrote a test case to get aws resource object for dynamodb and create table via resource object and assert if count of items in table is 0 (implies table was created).

2. test_dynamodb_hook.py
   adds mock_dynamodb2 from moto (https://github.com/spulec/moto). Wrote a test case to get aws resource object for dynamodb and create/insert data in dynamodb table. Then this unit test writes 10 items in the table using write_batch_data operation

3. HiveToDynamoDBTransferTest
   this unit test checks the workings of HiveToDynamoDB operator by inserting 1 record in dynamodb table via write operation.

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"